Limits of expertise in forecasting

Limits of expertise in forecasting is the distinction between genuine, domain-specific skill and the much broader claim that a person can reliably predict change or make superior strategic decisions. The book argues that expertise can be profound yet narrow: a specialist may recognize patterns within a well-defined field, but neither intelligence nor professional authority guarantees sound forecasts across uncertain problems. The practical conclusion is not to discard expertise, but to combine informed individual judgment with independent, diverse contributions and mechanisms that aggregate them.

What Expertise Can and Cannot Do

Expertise is unmistakable in fields where sustained practice creates specialized perception. A chess master does not merely calculate faster than an amateur: the master recognizes meaningful patterns in a game in progress and can reproduce its arrangement from memory. In demonstrations by Herbert A. Simon and W. G. Chase, experts could recall the layout of an actual chess position, whereas amateurs could not. But when pieces were placed irregularly, the experts lost that advantage. Their remarkable memory depended on a meaningful chess pattern, not on a general capacity to remember arbitrary arrangements.

That contrast defines the boundary of expertise. Chess skill does not automatically transfer to unfamiliar problems; expertise is “spectacularly narrow.” The same caution applies to broad labels such as decision making, policy, or strategy. Auto repair, piloting, skiing, and perhaps management may yield to practice, talent, and hard work. Forecasting an uncertain future and selecting the best course in response to it are different tasks: the evidence presented in the book gives little reason to assume that prolonged experience in a field makes someone reliably expert at forecasting change in general.

The narrowness matters because organizations often treat intellectual ability as portable. A person celebrated for solving one class of problems may be presumed capable of solving another, and the conspicuous confidence of a business leader can make that presumption persuasive. But success at recognizing patterns in a bounded domain is not proof of a general ability to see what comes next. The relevant question is not whether expertise exists, but whether the particular problem is one to which the expert’s practiced skill genuinely applies.

The Record of Forecasts and Judgment

Anecdotes about celebrated executives making implausible predictions can be dismissed as inevitable exceptions: over a century, even intelligent people will sometimes say foolish things. The more serious challenge is the broader record of professional forecasting and investment. Between 1984 and 1999, almost 90 percent of mutual-fund managers underperformed the Wilshire 5000 Index. In the most recent five-year period discussed in the book, more than 95 percent of managed bond funds also underperformed the market.

The results described across other fields are similarly sobering. After examining studies of expert forecasts and analyses, J. Scott Armstrong reported finding no studies that showed an important advantage for expertise. Experts sometimes forecast somewhat better than laypeople, while some studies found nonpsychologists better than psychologists at predicting human behavior. But beyond a minimal level of expertise, Armstrong concluded, expertise and accuracy were unrelated. James Shanteau, a leading researcher on expertise, likewise warned that expert decisions were seriously flawed.

Disagreement is one sign of the difficulty. In fields including stock picking, livestock judging, and clinical psychology, agreement between experts fell below 50 percent: experts were as likely to disagree as agree. Another measure, internal consistency, exposes a different problem. Medical pathologists judging the same evidence gave different opinions half the time; the reported consistency measure was 0.5. An expert’s title, then, does not ensure that a judgment will be reproducible, even when the evidence presented is unchanged.

Confidence creates a further gap between judgment and reliability. A well-calibrated forecaster understands how often judgments at a given level of confidence prove correct. Yet the professionals surveyed by Terrance Odean—including physicians, nurses, lawyers, engineers, entrepreneurs, and investment bankers—believed they knew more than they did. In a study of foreign-exchange traders, predictions about exchange rates were overconfident 70 percent of the time. The traders were not simply mistaken; they also misjudged how likely they were to be mistaken. The book identifies expert bridge players and weather forecasters as exceptions whose judgments are routinely well calibrated: when weather forecasters predict a 30 percent chance of rain, it rains on 30 percent of those days.

Armstrong’s broader conclusion is that expertise beyond a minimal level has little value for forecasting change. Nor does the evidence establish that a small class of exceptional forecasters can reliably overcome the limitations of the rest: claims of accuracy by a single expert have no practical value without a dependable way to distinguish skill from luck. This is not a claim that trained professionals lack knowledge. It is a warning against treating professional standing or past success as sufficient evidence of reliable predictions about an uncertain future.

Why Aggregated Judgment Can Be Stronger

The alternative is not uninformed guessing. The book’s account of collective judgment depends on four conditions: diversity of opinion, independence of judgment, decentralization that lets people use specialized or local knowledge, and aggregation that turns individual judgments into a collective decision. In a sufficiently large group, each estimate combines information with error. When the estimates are diverse and independent, individual errors can cancel in the average, leaving more of the information intact.

Cancellation alone cannot make an ignorant group wise. If a crowd has little relevant information, averaging its guesses does not supply what it lacks; a group of children trading stocks after the Challenger disaster would not be expected to identify Thiokol as responsible. The force of collective judgment lies instead in the possibility that many people hold different fragments of relevant information. The average can combine fragments that no single participant possesses in full.

Evidence from prediction markets illustrates the contrast between individual certainty and collective performance. In the National Football League betting market, point spreads only weakly predict final scores, but well-informed gamblers find it very hard to beat the final spread consistently. Favorites cover in roughly half of games, as do underdogs; there is no obvious, persistent mistake such as home teams winning more often than the crowd predicts. Though occasional patterns can be found, including a historical advantage for home underdogs in weeks fifteen through seventeen, the market’s final line is usually difficult to outperform. About three-quarters of the time, the Mirage’s final line is the most reliable forecast of NFL game outcomes available.

The Iowa Electronic Markets provide another comparison. Across 49 elections from 1988 through 2000, election-eve prices were off by an average of 1.37 percent in presidential elections, 3.43 percent in other U.S. elections, and 2.12 percent in foreign elections. The market generally outperformed major national polls, sometimes by months. Of 596 polls released across the presidential elections in that period, the IEM price on the day of each poll was more accurate three-quarters of the time. The market was small, with no more than about 800 traders, most of them men and a disproportionate, though shrinking, number from Iowa. Its participants did not represent the electorate, yet their forecasts of what voters would do were better than asking voters what they intended to do.

The comparison does not make collective judgment infallible. Sports spreads are weak predictors of exact scores, and election-market prices can be wrong. Their significance is that they aggregate information in a way that makes consistent outperformance difficult, and that often produces forecasts more accurate than those of individual experts or polls. The point is comparative, not absolute: group estimates can be wrong, but expertise alone offers no dependable guarantee of being right.

Diversity, Local Knowledge, and Organization

Cognitive diversity is especially important in small groups and formal organizations. Large markets tend to include people with different risk tolerances, time horizons, investing styles, and information; anyone able to participate can enter. A team, by contrast, must actively make room for different perspectives. Without that effort, a few biased members can exert disproportionate influence and distort a group’s decision.

Political scientist Scott Page tested this idea with computer-simulated groups of ten or twenty problem-solving agents, each with a different set of skills. Some agents were individually very good at solving a relatively sophisticated problem; others were less effective. Groups that combined high- and lower-performing agents almost always did better than groups made up only of the strongest agents. Selecting a group randomly and letting it work could do as well as, or better than, spending substantial time identifying the smartest agents and then assigning the task to them alone.

The result does not imply that intelligence is irrelevant or that an uninformed collection of people will outthink an expert. Page’s agents had skills, and successful groups included high performers. The point is that intelligence alone does not ensure that a group contains different ways of approaching a problem. If the most highly valued abilities cluster among similar people, a team of apparent superstars may collectively know less than a more varied group. A less accomplished participant can contribute a different skill that expands what the group can see.

Decentralization can protect this variety by allowing people close to a problem to draw on knowledge acquired through a particular job, place, or experience—what Friedrich Hayek called tacit knowledge. Its weakness is that local discoveries may never reach the rest of the system. A capable organization therefore needs both room for local specialization and an effective way to aggregate what individuals learn. Sharing information need not mean placing one small group at the center to interpret everything: the book’s treatment of the intelligence community before September 11 distinguishes aggregation from centralization. Agencies had specialized knowledge, but no effective mechanism to combine judgments across them; linking databases and circulating information could have improved collective awareness without erasing the value of independent expertise.

The same problem appears inside corporations. At Ford, more than fifteen management layers separated the chairman from a factory-floor supervisor, while Toyota had five. In one General Motors plant, two workers installed a bracket for two years without being told that it held the car’s sunshade; when a supervisor showed them how a missed bracket required the carpet to be torn out and the bracket welded into place, one worker was surprised to learn what the part did. The distance between a task and its purpose was also a distance through which useful information could be lost.

Hierarchy can make that distance worse. One company chart reproduced in In Search of Excellence showed 223 different paths a new product idea could take through the bureaucracy before acceptance. Meanwhile, studies of executives found that anxiety about promotion was associated with less accurate communication of problem-related information, and that upwardly mobile managers were less likely to tell bosses about failures. When dissent is risky, the organization may possess relevant knowledge without allowing it to shape a decision. The proposed remedy is not to eliminate expertise or coordination, but to give local knowledge and independent analysis room to move while maintaining ways to bring separate judgments together.

Using Experts Without Chasing Them

The argument rejects neither specialists nor their contributions. A crowd of amateurs should not collectively perform surgery or fly planes, and informed analysts can help a group make better decisions. But their advice and predictions should be pooled with those of others, so their information can contribute to a judgment that is not dependent on a single person’s perspective. The larger and more diverse the informed group, the more reliable its collective judgment is expected to be.

This makes the search for a lone answer-giver—“chasing the expert”—a poor organizational strategy. A group’s decision can outperform most of its members repeatedly, while an individual expert’s performance varies with the problem. Even if a few people can consistently outjudge an informed, diverse group, there is no easy way to identify them. Warren Buffett is offered as a possible example because he had consistently outperformed the S&P 500 Index since the 1960s. But a long record of apparent success still raises the problem of distinguishing skill from luck or measurement error, and past performance does not guarantee future results.

The limit of expertise in forecasting is therefore not a verdict against knowledge; it is a boundary on what a credential, a reputation, or a record can establish. Specialists can see deeply into the domains they know. The error is to confuse that depth with a general power to foresee change, or to assume that locating one superior mind is safer than building a decision process that collects many informed minds. Expertise has its greatest practical value when it enters that larger process, contributing knowledge without being asked to stand in for the whole.